In many digital imaging applications, light from a subject can be represented in an image. Despite improvements in sensor technology, the light signal or brightness range of naturally occurring and constructed subjects often exceeds the nominal output range of instruments available to detect photons. For this reason, numerous methods have been described in the art of photography to extend the dynamic or signal range of images. Typically, these approaches involve methods for combining data from different images into a composite image based on methods to approximate the relationship between the initial images. Composite images that represent data with extended dynamic range could provide significant advantages for the presentation and quantitative study of the relationships between elements within such a high dynamic range subject. Such composites facilitate rapid comparison of disparate intensity objects by visual inspection. Furthermore, such composites permit more compact storage of information than the aggregated source data.
The study of genomics, proteomics and like fields apply technologies to analyze all genes from an organism on a common reference basis. Often, these technologies employ fluorescent or radioisotopic tags to provide high sensitivity and dynamic range. Genome-wide analysis of yeast indicates that 80% of genes generate 10 or fewer mRNA molecules per cell. Moreover, genes of critical interest that regulate cellular activity often produce rare mRNAs. In the area of gene expression, the products (messenger RNA (mRNA) or protein) of a highly expressed gene may exceed those of another gene by a million-fold within a given cell. This range of gene expression extends beyond the typical 65,536-fold (16 bit) maximum capabilities of high performance scientific detectors. In addition, the nominal output data of the sensor includes some amount of noise that reduces the useful signal range. For scientific imaging, the usable signal above the noise level provides a more relevant measure of performance than nominal output data range.
Extended Dynamic Range
The desire to extend dynamic range in images is not unique to scientific measurement and numerous methods have been described to address this issue in the art of photography. These approaches have several shortcomings relative to processing and analyzing scientific images. U.S. Pat. No. 6,040,858 to Ikeda (hereinafter referred to as “Ikeda”) describes two general approaches to image combination for extending dynamic range. While these methods may generate visually pleasing images, they do not address the demands of scientific imaging. In both approaches, a properly exposed image that does not have saturated pixels is used as the “standard image.” The standard image is combined with a rescaled version of a nonstandard image in one method. Below a brightness threshold, values from the standard image are used; above this threshold, values are computed from the under-exposed image. To accommodate the extended dynamic range, high values are compressed to fit within the standard output data representation. This method introduces a non-linear relationship between output data from the sensor system and the light emitted by the subject. Specifically, the brightest objects have disproportionately lower output values that would be expected from the less bright objects. Losing the linear relationship between object brightness and output data value limits this method's application to scientific quantification.
The other prior art method described by Ikeda combines a standard image with a different non-standard image. In this approach, the non-standard image contributes a higher output noise value than present in the standard image. To mitigate this noise value, a threshold is set above the noise value and the combined image is constructed from values above this threshold. Thus, the potential for extended dynamic range is limited by the scaling of noise in the composite image.
FIG. 1 diagrams the problem of increased output noise level in the resultant composite image resulting from component image rescaling in prior art approaches. FIG. 1 shows the signal data bits as open boxes and noise data bits as shaded boxes. Specifically, Source Data A 10 and Source Data B 20 are two corresponding pixels in 14-bit output data format that will be combined into one Composite Data 16-bit 30 output data format pixel. The data in these images can be scaled in proportion to the System Gain 40 used to capture each image. However, if noise accounts for the lowest 4 bits within each source pixel the resulting image will still only have about 10 bits of signal above noise 50. Under these conditions, there is little or no significant gain in dynamic range over the initial noise. Applying a threshold as described by Ikeda improves the signal to noise ratio in the thresholded data 60 (2 bits of noise in 12 rather than 4 in 14), but does not substantively improve the quantifiable signal range over the Source Data A 10.
Sensor Attributes
Many types of photon detecting sensors are known in the art, these include: charge-coupled device (CCD), complementary metal oxide semiconductor (CMOS), amorphous silicon, passive and active pixel sensors, photomultiplier tubes (PMT), microchannel plates, vidicon tubes, and photodiodes. Sensors can be classified as point sensors and field imaging (two dimensional or array) sensors depending on whether they have one or a two dimensional array of photon detecting elements. Both photon detecting elements and the data they produce may be called pixels for “picture elements.” It is well known in the art that point sensors can be scanned to produce two dimensional images. Thus, these various sensors can be discussed together.
The sensor absorbs light signals from the subject, transforms them into electrons and the electrons are converted into output values or digital numbers (DN). A field sensor images the spatial locations on the subject into pixels, and assigns each pixel a DN for total signal received. The maximum number of electrons that a pixel can hold limits the signal range of the sensor. In turn, this limit corresponds to a certain number of signal values it can assign each pixel (0 to 4096 for a 12-bit sensor). The saturation point (SSat) is the limit beyond which the sensor can record no more signal (4096 for a 12-bit sensor). The minimum signal (SMin) is the minimum signal required for the sensor to record a signal value (1 for a 12-bit sensor).
A number of factors affect the relationship between the amount of light received by a sensor and its DN output. It is well known that various sensors differ in their Quantum Efficiency (QE). QE varies with respect to wavelength and is the number of electrons produced per number of photons received with respect to a theoretical ideal. To convert the electrons to a DN, sensors employ analog to digital converters (ADC). The relationship between input voltage levels and digital number output may also be called gain. Photon counting sensors may also be characterized by their “linear response range.” Within this range, the signal detected for each pixel (S) is directly proportional to the light from the subject received by the sensor. SSat represents the maximum limit for the linear response range.
After reaching SSat for a certain pixel, an ordinary sensor is not only unable to record more signal, but is also unable to contain the excess electrons. Therefore any additional signal received at that pixel will result in electrons overflowing to unsaturated neighboring pixels. This creates falsely high apparent signals for those neighboring pixels. When observed with 2-dimensional CCDs and other field imaging sensors, this phenomenon is commonly called “blooming.” The specific shape of blooming reflects the path of excess electron overflow based on sensor architecture rather than the subject. Since the overflow signal contaminates the target signal, data is lost when blooming occurs.
Various methods are known to overcome the limitation of signal overflow. Some sensors are equipped with ‘anti-blooming’ technology that prevents electrons flowing into neighboring pixels even at signal levels well beyond saturation. FIG. 2 compares signal ranges for an ordinary sensor with an anti-blooming sensor. Here, the signal from an ordinary sensor 70 shows that the point of signal saturation (Ssat) 75 and the point of signal blooming or overflow (Sbloom) 75 are the same. Therefore, any signal not in the recorded signal 72 overflows or “blooms” into neighboring pixels, showing here as blooming 74. An anti-blooming sensor 80 signal has separate saturation (Ssat) 81 and blooming (Sbloom) 83 points such that the portion of the signal that is not part of the recorded signal 82 does not overflow into neighboring pixels. The additional signal 84 goes unrecorded, neighboring unsaturated pixels remain unaffected, and the unsaturated pixels in the image will contain reliable signal values that are proportional to the light emitted by the subject. Eventually, anti-blooming sensors also reach a signal level where electrons overflow to adjacent pixels and blooming 86 occurs beyond the blooming point (SBloom) 83, but this can be several times Ssat 81.
Technology Trends
Two additional factors deserve note. First is the exponential increase in available performance from computers with more RAM, faster processors, fixed and removable disk storage. This trend reduces barriers to applying image processing solutions that may be computationally intensive to increasingly high resolution images. Generally, methods that can be implemented as software solutions can provide for improved, lower cost support of multiple apparatus architectures.
Second is the growing market for low cost, high resolution consumer-oriented digital cameras. This trend results in CCD and CMOS detectors with many more numerous, but smaller pixels that have reduced electron well capacity. This decreased well capacity coincides with decreases in effective dynamic range and signal to noise ratio for consumer grade sensors. By combining appropriate dynamic range extension methods with consumer grade sensors, these inexpensive sensors may become more useful for a broader range of applications including scientific applications.